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ABSTRACT 

The National Center for Research in Vocational 
Education (NCRVE) partially supported 10 new or extensively revised 
leadership development projects at 7 institutions of higher education 
designed to improve the leadership attributes of graduate students in 
vocational education. The projects adopted as instructional 
objectives one or more of the leadership attributes proposed by NCRVE 
and cooperated in implementing a common evaluation plan. From 180 
participants and project directors, NCRVE collected quantitative data 
about participants' preprogram leadership attributes, satisfaction 
with the program, postprogram attributes, and leadership behavior and 
performance 6 months after the program as well as qualitative data 
about perceptions about effectiveness of program activities. 
Meta-analysis results indicated the following: participants were very 
satisfied with the programs and believed they were of great value to 
their professional development; overall, the programs had a 
significant impact on the participants* perceptions of their leader 
attributes; the number of leader attributes that improved 
significantly appeared related to certain program characteristics; 
programs were not very successful in developing prespecified 
attributes; and of the six attributes that best explained leader 
effectiveness, two (adaptable, open to change and insightful) readily 
improved and two (team building and willing to accept responsibility) 
were more resistant to change. (YLB) 
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BACKGROUND 



One goal of the National Center for Research in Vocational Education (NCRVE) at 
the University of California at Bcirkeley is to increase the number and quality of leaders 
prepared to meet present and futun^ challenges facing vocational education. NCRVE seeks 
to accomplish this general purpose by stimulating, facilitating, and evaluating educational 
interventions which affect positive change in selected "leader attributes." These leader 
attributes are the characteristics, knowledge, and skills of individuals which increase the 
likelihood that they will (1) perceive opportunities to behave as leaders, (2) grasp those 
opportunities, and (3) succeed in influencing group behaviors in a wide variety of 
situations and professional roles. 

Toward this end, NCRVE partially supported new or extensively revised leadership 
development projects at seven institutions of higher education designed to improve the 
leader attributes of graduate students in vocational education L The projects took multiple 
forms, varying in approach and length (e.g., courses, workshops, and internships). Each 
adopted as instructional objectives one or more of the leader attributes proposed by 
NCRVE, and each agreed to cooperate in implementing a common evaluation plan. The 
responsibility of NCRVE has been to evaluate the effectiveness of the projects so that more 
can be learned about the strategies and activities for developing leader attributes. 



EVALUATION QUESTIONS 

The nine key evaluation questions were as follows: 

1 . What types of leadership activities have been developed and implemented by 
cooperating universities? 

2. How many, and what types of people participated in the leadership development 
programs? 



^Thc seven institutions were Colorado State University (conducted two programs), Indiana State University, 
Iowa State University, Mississippi State University (conducted two programs), North Carolina State 
University, University of Georgia, and University of Maryland. In addition* a state supported program at the 
University of Minnesota agreed to be evaluated. This totals ten programs. 
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3. What were the costs of different leadership programs? 

4. How satisfied were participants with the various leadership development programs? 

5. To what degree did participants perceive a change in their leader attributes as a 
result of participating in the leadership development programs? 

6. To what extent did the leadership development activities affect how participants 
perceive their ability to behave and perform as leaders? 

7. What kinds of impact did the leadership development program have on the 
institutions' involvement in developing and maintaining leadership learning 
activities for their students? 

8. What activities were considered particulariy effective and what leader attributes did 
they impact? 

9. What recommendations can be made for improving leadership development 
activities? 



PERSPECTIVE 

Even a cursory review of the literature on leadership reveals that a great deal has 
been written by authors from a wide variety of disciplines and fields of practice. Despite all 
the attention (or perhaps because of it), the topic presents a conceptual snakepit. There is 
no agreement about what leadership is, why it occurs, how it is best developed, or how it 
should be assessed. Fortunately, there is consensus that behaviors can be related to 
performance as a leader, and that educational interventions can affect the behavior of 
leaders. Consequendy, the first task of NCRVE was to create its own conceptualization of 
leadership and leadership development (Moss & Liang, 1990) and its own instrument, the 
Leader Attributes Inventory (LAI), for assessing leadership development (Moss, Johansen, 
&Preskill, 1991). 

At the heart of the conceptualization are thirty-seven leader attributes which are 
hypothesized to be positively related to effective performance as a leader — especially a 
leader in vocational education. Three studies have been completed to date by NCRVE 
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which support that hypothesis (Moss & Johansen, 1991). Using the LAI instrument, 
which contains the thirty- seven attributes, correlations between each of the attributes ajid. 
subordinates' perceptions of leader effectiveness were found to range from .56-. 82 
(averaging .70); a multiple correlation coefficient of .90 was attained by combining six of 
the thirty-seven attributes: (1) motivating others; (2) team building; (3) adaptable, open, 
flexible; (4) gathering and managing information; (5) willing to accept responsibility; and 
(6) insightful. 

Given some empirical evidence supporting the validity of the leader attributes, a 
conference was held in 1990 for teacher educators in graduate programs of vocational 
education to familiarize them with NCRVE*s concept of leadership and with promising 
leadership development strategies and activities. Following the conference, a "Request for 
Proposals" inviting proposals to design and conduct leadership development activities for 
graduate students was sent to all vocational teacher education departments offering graduate 
programs. The programs evaluated by NCRVE are those which were selected from among 
the applications received, funded, and subsequently conducted. 



EVALUATION DESIGN 

Table 1 presents the means of data collection used to answer each of the nine 
evaluation questions previously noted. Data was collected from participants and project 
directors. Some of it was quantitative; some of it was qualitative. Quantitative data 
provided information about the preprogram leader attributes of participants, their 
satisfaction with the program, their postprogram attributes, and their leadership behavior 
and performance six months after the program was concluded. Qualitative data included 
directors' and participants' perceptions about the effectiveness of various program 
activities. 
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Table 1 

Evaluation Questions and Means of Data Collection 



Key Questions 


#1 


#2 


#3 


#4 


#5 


#6 


#7 


#8 


#9 


Participant 
Satisfaction Survey 








X 








X 


X 


Leader 

Attributes Inventory 




X 






X 






X 


X 


Performance Survey 












X 






X 


Faculty Interviews 


X 


X 


X 








X 




X 


Participant Focus Group 


X 






X 


X 






X 


X 


Cost 






X 














Participant Description 




X 
















Faculty Description 




X 
















Activity Description 


X 














X 


X 



Guided by the nine evaluation questions, each program (project) was analyzed 
separately. Tlien, the findings about the ten programs were examined to derive insights 
useful to others who may be planning to conduct their own leadership development 
programs. In this meta-analysis stage, the units of analysis were most frequently programs 
(n=10), but attributes (n=37) and individuals; (n=180) were also used. Statistical 
techniques for combining, comparing, and relating data were utilized. 



PROGRAMS 

The characteristics of the ten leadership development programs provided by the 
eight institutions varied considerably: 

• They ranged in length and intensity from a total of six hours in one day to nmety 
hours of class instruction plus one-hundred eighty hours of outside assignments 
spread over a nine-month period. 
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The number of students varied from four to twenty-five per class section with a 
mean of sixteen. 



Key features included (1) seminars with a semester-long internship; (2) seminars 
coupled with field trips (one to five days each); (3) seminars plus teams of 
participants instructing teachers in the field; (4) one-day workshops focused on 
health-related attributes; (5) seminars with a focus on self-assessment and planning 
for self-improvement; (6) three, two-and-a-half- to five-day retreats witii a couple 
of months between sessions; and (7) team taught seminars with applications to 
contemporary problems in vocational education. 

The number of attributes chosen as instructional objectives by each program langed 
from four to twenty-two. 

The attributes selected as insutictional objectives by five or more programs included 
communication; visionary; confident, accepting of self; networking; and team 
building. Four of the thirty-seven attributes were not used as objectives by any of 
the programs: (1) accountability, (2) intelligent with practical judgment, (3) 
personal integrity, and (4) ethical. 

The average program cost per student (excluding indirect costs) was $1409, 
ranging from $174 to $4378. 



PARTICIPANT CHARACTERISTICS 

Table 2 presents a description of the one-hundred eighty students who participated 
in the ten programs. 
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Table 2 
Participant Characteristics 



Characteristic 


Percent 


Full-time student 


39 


Part-time student 


61 


Doctoral 


61 


Other 


39 




49 


reuiaie 


51 


Cauca5»ian 


00 


Other 


14 


Experience as a school administrator 


47 


No experience 


53 


Experience as a nonschool manager 


68 


No experience 


32 


Thirty-five years or less 


27 


Thirty-six years and over 


73 



LIMITATIONS 

Like any study, this one has limitations. These should be n.ade explicit before 
describing the results of the evaluation. 

First, the sample of programs was small. Only ten programs were evaluated and, 
in the case of six-month follow-up data, only nine programs participated. 

Second, no experimental controls were exercised. Given the present lack of 
knowledge about how leadership development programs can best be delivered, it was 
decided to encourage programs that used a wide variety of approaches. This necessitated 
structuring the evaluation as an exploratory rather than a confirmatory study. That is, the 
focus is on searching for relationships that might later be tested by more rigorous 
(experimental) designs. 
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Third, all the data on outcome variables was based upcn participant self-perceptions 
and self-reports of activities. They were appropriate measures, but the credibility of the 
results would have been enhanced had it also been possible to secure the views of 
observers about participant behaviors. 

Fourth, the follow-up period of six months is relatively short, albeit as long as 
circumstances permitted. 

Given the limitations, the analysis of the data did not attempt to test a priori 
hypotheses to reach general conclusions. Instead, the intent of the meta-analysis was to 
explore the data relevant to each evaluation question so as to identify, describe, and 
summarize results in a way that would be maximally useful to those who will plan and 
conduct future programs of leadership development. 



HIGHLIGHTS FROM THE RESULTS 

The following highlights have been abstracted from the results of the meta-analysis 
of the ten leadership development programs: 

1 . Based upon both qualitative and quantitative data, participants were very satisfied 
with the programs and believed that they were of great value to their professional 
development: 

• The average rating of the ten programs on the question of "value of the 
experience" was 4.6 (five-point scale). 

• Participants not only felt that similar programs should be made available to 
other graduate students, but they also wanted the programs to be lengthened 
or additional programs provided. 

2. Overall, the ten programs had a significant impact on the participants' perceptions 
of their leader attributes: 

• Of all possible increases in attribute scores on the thirty-six percent 
had statistically significant (p<.05) gains. 



Post minus pre (retrospective) scores on each of the attributes in ten different programs. 
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• A major effect of the programs was to lower participants' perceptions of 
their attributes as they were before the program (retrospective scores). 

• The distributions of post (ai'ter treatment) attribute scores were much more 
compressed around a higher point value than were the pre (retrospective) 
scores. This effect was demonstrated by significant (p<.05) negative 
correlations between pre (retrospective) scores aiid gain scores on all thirty- 
seven attributes (r=-.65 to -.85). 

The number of leader attributes that improved significantly appears to be related to 
certain program characteristics: 

• Readiness for change was developed by providing experiences for team 
building and assessing participants' attributes, with time for reflection and 
goal setting. 

• The more focused and structured the experience, and the more active the 
involvement of the student, the more likely it was for attributes to improve. 

• The number of hours of directly supervised instruction was positively 
related (p<.05) to the number of attributes that increased significantly 
(rs=.36). 

There was a positive relationship between program cost per student and hours of 
directly supervised instruction (rs=.68): 

• A large portion of program cost was the instructor's salary, which varied 
according to the amount of time devoted to the program. 

• The relationship between cost per student and the number of attributes 
which increased significantly was not significant. Factors other than 
program length apparently contributed to program effectiveness. (See item 
3, above.) 

Programs were not very successful in developing prespecified attributes: 

• Just forty-one percent of the attributes specified as instructional objectives 
by the programs made significant (p^.05) gains. 

• Of all the significant attribute gain scores, otily thirty-four percent were 
prespecified as instructional objectives; sixty-six percent were not 
prespecified as instructional objectives. 

• There are several possible reasons for this result: (1) individuals were given 
choices in reading and other assignments and thus were exposed to different 



experiences; (2) individuals inevitably learn different things from the "same" 
instruction as they interpret the experience la terms of their own values and 
cognitive structures; (3) the attributes may not be entirely independent; and 
(4) program designers do not yet know the best way to develop specific 
attributes. 

Improveirr.nts in leader attributes were not meaningfully related to any of the 
participant characteristics measured as a part of the evaluation. 

• Only three percent of the possible differences in LAI gain scores between 
the thirty-five years and under and the thirty-six years and over age groups 
were significant (p<.05). 

• Just four percent of the possible differences in LAI gain scores between the 
participants with experience as school administrators and the participants 
without that experience were significant. 

• About six percent of the possible differences in LAI gain scores between the 
participants with experiences as nonschool managers and the participants 
without thatexperience were significant. 

• Only four percent of the possible differences in LAI gain scores between 
women and men were found to be significant. 

• About five percent of the possible differences in LAI gain scores between 
participants pursuing the doctorate and those with other degree objectives 
were found to be significant. 

• Just four percent of the possible differences in LAI gain scores between 
participants who were part- and full-time students were found to be 
significant 

Of the six attributes ihat best explain :eader effectiveness (Moss & Liang, 1990), 
two were readily improved^ and two appeared more resistant to change"^: 

• The readily changeable attributes were adaptable, open to change; and 
insightful. 

• The more resistant attributes were team building, and willing to accept 
responsibility. 



Included as instructional objectives in three or more programs and significantly increased in sixty percent 
or more of the programs. 

Included as instructional objectives in four or more programs, but significantly improved in only twenty- 
five percent or less of those programs. 



• Table 3 shows this relationship. 

All thirty-seven leader attributes were used by participants during the six-month 
period following instruction: 

• Between fifteen and fifty-six percent of the respondents considered each of 
the leader attributes "most useful " 

• Six attributes were considered most useful by fifty percent or more of the 
respondents: (1) insightful; (2) adaptable, open to change; (3) visionary; (4) 
willing to accept responsibility; (5) communication (listening, oral, written); 
and (6) team building. 

The attributes considered "most useful" by the highest percentage of the participants 
tended to be the attributes that best explained effective leader performance: 

• Four of the six attributes considered most useful by fifty percent or more of 
the respondents were also among the group of six attributes that best 
explained the variation in leader effectiveness (per the 1990 study by Moss 
and Liang). These were (1) adaptable, open to change; (2) insightful; (3) 
willing to accept responsibility; and (4) team building. 

• Table 3 shows this relationship. 

Six months after the programs, participants felt that the experience had, on the 
whole, contributed a "fair amount" to their successful performance as leaders: 

• The mean rating on a four-point scale was 2.8. 

• Only half of the six "most useful" attributes were found to be readily 
improved by the leadership programs: (1) adaptable, open to change; (2) 
insightful; and (3) visionary. Thus, participants gained only a "fair amount" 
from the programs (see Table 3), 

An average of fifty-seven percent of the participants in each program engaged in a 
greater number of leadership activities during the six-month period following 
instruction than they had engaged in before the instruction: 

• There were no differences among the programs in the percent of participants 
who engaged in a greater number of leadership activities. 

• Forty-five percent of the additional leadership activities were job-related, 
twenty-four percent were in new professional roles, twenty-one percent 
were community activities, and ten percent were social. 
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Table 3 

Relationships Among Attributes: Ability to Explain Performance, 
Usefulness, and Readiness to Change 



Leader Attribute 


Best Explain 

LeacJer 
Performance* 


Most 
Useful to 
Participants 


Readiness to Change 


Most 
Readv 


Most 
Resistant 


Adaptable, open to change 


X 


X 


X 




Insightful 


X 


X 


X 




Willing to accept responsibility 


X 


X 




X 


Team building 


X 


X 




X 


Motivating others 


X 








Gathering and managing information 


X 








Visionary 




X 


X 




Communication 




X 




X 



* From Moss & Liaag, 1990. 



12. Eighteen percent of the pmicipants reported engaging in additional leadership 
training activities during the six-month period following instruction: 

• There were significant differences among programs in the percent of 
students who engaged in further training, but these percentages were not 
related to hours of supervised instruction or number of attributes that were 
significantiy improved. 

• The kinds of further training reported were primarily focused readings, 
courses, and workshops. 

1 3 . The programs appear to be having important and desirable institutional impacts: 

• Four institutions were adding new leadership courses to their regular 
graduate curriculums, and three other institutions were revising existing 
graduate or undergraduate courses to include greater emphasis on leadership 
development 

• A number of noncurricular outcomes were also reported by directors (e.g., 
initiating an institution-wide study of leadership development 
opportunities). 



ERIC 



14. Certain kinds of program activities appear to be effective in improving certain leader 
attributes: 

• Self-assessment (with planning) improved the attributes of confident, 
accepting of self; adaptability, open to change; and appropriate use of 
leadership styles. 

• Observations and interviews of leaders at work improved the attributes of 
visionary, and courageous, risk-taker. 

15. LAI is a useful assessment tool: 

• LAI was sensitive to the effects of treatments on attributes (means and 
distributions). 

• It was not sensitive to differences in age, experiences as a school 
administrator, experiences as a nonschool manager, gender, academic 
status, or part- or full-lime participation in a degree program among 
graduate students. 

• The attributes it assesses were used by participants in their leadership 
activities. 



RECOMMENDATIONS 

Based upon the results of the meta-analysis, the following rec ndations seem 
warranted: 

I . The success of the ten leadership development programs justifies implementing a 
greater number of programs for graduate students in vocational education, and 
continuing research and development to improve their effectiveness. 

Leadership development programs for graduate students should include the 
following characieri:>tics: 

• Careful course structure and direction by the instructor are necessary to keep 
participants focused on the program objectives. 

Participants should be helped to construct a cognitive model of leadership 
that can guide their further leadership development. Readings, 
presentations by role models, and "shadowing" experiences are helpful in 
this regard, 

18 
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Team building experiences should be provided early in the program to build 
a safe, supportive environment in which attribute changes are encouraged 
and facilitated. 

At the same time, a number of self-assessment instruments (inventories and 
tests) of leader qualities and styles should be administered to (1) sensitize 
participants to their weaknesses as a basis for improvement and to their 
strengths as a foundation for building upon; and (2) help participants 
understand, respect, and appreciate behavioral differences among 
individuals. 

Opportunities to plan for self-improvement, based upon self-assessment, 
are useful mechanisms to encourage reflection and goal setting. 
Sufficient time must be allowed for guided practice in applying the attributes 
to be changed and for reflecting on the experience. Simulations, exercises, 
games, and field assignments are useful tools. The time allocated to practice 
seems to distinguish between programs that teach about leadership and 
those that bring about behavioral modifications. 
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